Description

The SATerm Database Interface^? (aterm-dbi) offers an ATermServices look on a relational database. aterm-dbi uses Java Stratego.ATerm Servlets.

Interface

Currently there is just a query service. It is accessible on the URL aterm-dbi/query. The query service is an ATermService . It accepts a Query term in the body of a HTTP POST request and returns the result of the query in the body of the reponse.

For example this query:

  Query("SELECT ID, Name FROM Customer")

Might result in this ResultSet :

  ResultSet(
    MetaData(
      Columns(
        [ Column(Name("id"),   Type(Std("INTEGER"), Db("int4")))
        , Column(Name("name"), Type(Std("CHAR"),    Db("bpchar")))
        ]
      )
    )
  , Data(
     [ Row([160, "Meindert Kroese"])
     , Row([159, "Steven van Dijk"])
     , Row([158, "Martijn Schrage"])
     , Row([157, "Sandor Spruit"  ])
     ]
    )
  )

Invocation

From Stratego you can invoke the services of aterm-dbi with the http-transform strategy of StrategoNetworking.

There is one problem: the Java implementation of the ATermLibrary doesn't provide a reader for BAF, the Binary Stratego.ATerm Format^?. Because of this format is very efficient, it is used by XTC by default. You should thus invoke the services of aterm-dbi in TAF, the Textual Stratego.ATerm Format^?. You could do this with this strategy:

  http-text-transform(service-url) =
    xtc-temp-files(
      write-to-text
    ; xtc-http-transform(service-url)
    ; read-from
    )

Invocation of the query service of aterm-dbi:

  <http-text-transform(!URL("http://127.0.0.1:8080/aterm-dbi/query"))>
      Query("SELECT ID, Name FROM Customer")

Download

You can download the latest release at:

• http://losser.st-lab.cs.uu.nl/~mbravenb/software/aterm-dbi/

The war file is a Java Web Archive, a binary distribution. You can immediately put this file in your servlet container. The .tar.gz files are source distributions. You need Ant to build the sources.

You can also get the latest sources from the Subversion repository:

  svn checkout %SVNSTRATEGOXT%/trunk/experimental/aterm-dbi

In the samples package of Stratego Networking^? you can find some samples in the xmpl/aterm-dbi directory (unreleased, use the Subversion repository).

Implementation

aterm-dbi is a J2EE application. It uses a JDBC DataSource that is made available by the application server it is hosted in via the JNDI. Advantages of this approach:

• by using techniques developed for database servers that should be able to handle a lot of requests, the Stratego.ATerm Database Interface scales very well.

• You don't need to modify any Java code to use a different database managament system, or one at a different location.

Installation

You should be able to run the Stratego.ATerm database interface on any J2EE Application Server and using any database system implementation (as long as there is a JDBC 3 driver available).

I will provide instructions for a configuration where I'm using Tomcat 5 and PostgreSQL. Please consult the documentation of the application and database server if you choose other software.

Basic applications

First of all you need a PostgreSQL database. You need version 7.2 or 7.3. Please refer to the documentation of PostgreSQL if you need to install the database system by yourself.

You need a Java 2 SDK to run Tomcat. It is wise to use a recent release (version 1.4.x). Make sure the environment variable JAVA_HOME points to the location where you installed your J2SDK.

Next, you need Tomcat 5. Tomcat 4 will not do the job! Installation of Tomcat 5 is very straightforward nowadays.

Configuration and extension

Now the basic components of Java based database application are available, now comes the Stratego.ATerm database interface specific part.

You need a JDBC3 driver for your DBMS, in this case the JDBC3 driver for PostgreSQL. Put it in the common/lib directory of your Tomcat installation.

Now it's finally time to install the web archive (.war) of aterm-dbi. You should put this war in the webapps directory of Tomcat.

We still have to configure Tomcat with the database you want to use. This is application server and database specific, so you will probably need to consult some other documentation.

In Tomcat you should add this to the Host element with the name attribute localhost:

  <Context path="/aterm-dbi" docBase="aterm-dbi.war" debug="5" reloadable="true">
    <Logger className="org.apache.catalina.logger.FileLogger"
               prefix="aterm-dbi-log."
               suffix=".txt"
            timestamp="true"/>

    <Resource name="jdbc/aterm-dbi"
              auth="Container"
              type="org.postgresql.jdbc3.Jdbc3PoolingDataSource"/>

    <ResourceParams name="jdbc/aterm-dbi">
      <parameter>
        <name>factory</name>
        <value>org.postgresql.jdbc3.Jdbc3ObjectFactory</value>
      </parameter>

      <parameter>
         <name>databaseName</name><value>.....</value>
      </parameter>

      <parameter>
         <name>serverName</name><value>127.0.0.1</value>
      </parameter>

      <parameter>
        <name>user</name><value>....</value>
      </parameter>

      <parameter>
       <name>password</name><value>.....</value>
      </parameter>
    </ResourceParams>
  </Context>

We've chosen the PoolingDataSource implementation of the PostgreSQL JDBC driver, but if your application server offers a connection pooling implementation which interfaces with a ConnectionPoolDataSource, you should choose that option.

Todo or might do

• Implement services for inserts, updates and deletes. Maybe use terms instead of concrete syntax in some cases.

• Convert more SQL data types like bit, boolean, dates etc.

• Conceptually it is much more attractive to send an AST representation of the SQL statements to the ATermService instead of concrete syntax for SQL in an Stratego.ATerm wrapper. This would be an excellent combination with the use of ConcreteSyntax for SQL in Stratego. This would at compile-time ensure syntactical correctness of the SQL statements. There are some problems:
  • Many SQL statements use database specific constructs. The practical use of the Stratego.ATerm Database Interface^? would decrease if we would enforce standard SQL.
  • We need a PrettyPrinter on the server-side, because the database system (well, JDBC) just accepts concrete syntax for SQL.
  • We need a SQL syntax definition (but a starting-point is already available)

-- MartinBravenboer - 10 Mar 2003

Introduction

The ATerm (Annotated Term) Format is a format for exchanging structured data between tools. The ATerm format is a generic internal and external representation of data by means of simple prefix terms.

Abstract Data Type

An ATerm is an expression according to the following grammar:

  t  := bt                 -- basic term
       | bt { t }          -- annotated term

  bt := C                  -- constant
       | C(t1,...,tn)      -- n-ary constructor
       | (t1,...,tn)       -- n-ary tuple
       | [t1,...,tn]       -- list
       | "ccc"             -- quoted string
       | int               -- integer
       | real              -- floating point number
       | blob              -- binary large object

Format

ATerms can be exchanged in several formats:

• The Binary ATerm Format (BAF) is an efficient format that preserves maximal sharing.
• The Textual ATerm Format (TAF) preserves maximal sharing but is not as efficient as BAF.
• The plain text format (TEXT) is a textual format that does not preserve maximal sharing.

The baffle tool in the ATerm Library can be used to convert the formats.

Library

The ATerm Format is supported by ATerm Libraries for C, Java, and Haskell.

Application

The ASF+SDF Meta Environment operates on AsFix representations of source code in the ATerm format.

The Stratego program transformation language uses ATerms to represent abstract syntax trees:

• Specification defines transformation of ATerms
• ATerm library is used in implementation of terms
• Exchange of syntax trees as ATerms

Publications

The ATerm format is described in the following publication:

• M. G. J. van den Brand, H. A. de Jong, P. Klint, and P. A. Olivier. Efficient Annotated Terms. Software - Practice \& Experience, 30:259-291, 2000.

C

• Implements garbage collection of ATerms
• Ensures maximal subterm sharing
• Supports the efficient Binary ATerm Format (BAF) in which complete sharing of subterms is maintained.

Visit the official homepage for more information and downloads.

Java

• Ensures maximal subterm sharing.
• Does not support the Binary ATerm Format (BAF)

As far as I know there is no offical homepage for the ATerm library for Java. The software can be downloaded from the package base at the CWI. The name of the package is aterm-java.

Haskell

• official homepage

Scheme

• http://planet.plt-scheme.org/#aterm.plt

Publications

• Efficient Annotated Terms

• is accessible at some URL using HTTP
• takes an Stratego.ATerm^? input in the body of a HTTP POST request
• returns an Stratego.ATerm^? output in a HTTP response

Library support

You can implement an ATermService using any language. Packages that support the implementation of ATermServices:

• stratego-net
• Java Stratego.ATerm Servlets^?

Implementations

• Stratego.ATerm Database Interface uses Java Stratego.ATerm Servlets^?

-- MartinBravenboer - 10 Mar 2003

Summary

Converts an ATerm to a comparable XML document.

Description

The tools aterm2xml and xml2aterm support the conversion from ATerm to XML and vice versa. Since applications have different needs, there are three conversion modes available: implicit, explicit, and very explicit.

• The explicit mode is the default mode and supports a roundtrip for almost all ATerms (that is, an ATerm can be converted to XML and back without changing its structure).

• The implicit mode does not support such a roundtrip, but the XML is usually more attractive. Use this mode if you only want to export some ATerm to an XML application. The name 'implicit' is related to the more implicit structure in the resulting XML.

• The very explicit mode supports a roundtrip for all ATerms. This mode is the way to go if you need the guarantee that a roundtrip preserves the structure of all your ATerms.

The structure of the XML documents in the very explicit mode is generic: there are no language specific elements in these XML documents. The structure is described a RELAX NG schema.

Future Work

In a future release we will add support for a round trip in the implicit mode, based on schema/rtg information. This has already been implemented, but needs to be updated and refactored.

Examples

The following invocations illustrate how to invoke the tools:

$ aterm2xml -i foo.trm                  # convert to explicit xml
$ aterm2xml -i foo.trm | xml2aterm      # convert to explicit xml and back
$ aterm2xml -i foo.trm --implicit       # convert to implicit xml
$ aterm2xml -i foo.trm --very-explicit  # convert to very explicit xml

The following tables explain the differences between the three explicity modes that are support by aterm2xml.

Explicit XML

foo

<foo xmlns:at="http://aterm.org"/>

foo

foo(1)

<foo xmlns:at="http://aterm.org">
  <at:int>1</at:int>
</foo>

foo(1)

1

<at:int xmlns:at="http://aterm.org">1</at:int>

1

"abc"

<at:string xmlns:at="http://aterm.org">abc</at:string>

"abc"

()

<at:tuple xmlns:at="http://aterm.org"/>

()

(1, 2)

<at:tuple xmlns:at="http://aterm.org">
  <at:int>1</at:int>
  <at:int>2</at:int>
</at:tuple>

(1,2)

[]

<at:list xmlns:at="http://aterm.org"/>

[]

[1, 2]

<at:list xmlns:at="http://aterm.org">
  <at:int>1</at:int>
  <at:int>2</at:int>
</at:list>

[1,2]

fred([foo, bar])

<fred xmlns:at="http://aterm.org">
  <at:list>
    <foo/>
    <bar/>
  </at:list>
</fred>

fred([foo,bar])

fred(None, [foo, bar])

<fred xmlns:at="http://aterm.org">
  <None/>
  <at:list>
    <foo/>
    <bar/>
  </at:list>
</fred>

fred(None,[foo,bar])

fred(Some(barney), [foo, bar])

<fred xmlns:at="http://aterm.org">
  <Some>
    <barney/>
  </Some>
  <at:list>
    <foo/>
    <bar/>
  </at:list>
</fred>

fred(Some(barney),[foo,bar])

fred("foo", "bar")

<fred xmlns:at="http://aterm.org">
  <at:string>foo</at:string>
  <at:string>bar</at:string>
</fred>

fred("foo","bar")

foo{fred}

<foo xmlns:at="http://aterm.org">
  <at:anno>
    <fred/>
  </at:anno>
</foo>

foo{fred}

Very Explicit XML

foo

<at:appl xmlns:at="http://aterm.org" at:fun="foo"/>

foo

foo(1)

<at:appl xmlns:at="http://aterm.org" at:fun="foo">
  <at:int>
    <at:value>1</at:value>
  </at:int>
</at:appl>

foo(1)

1

<at:int xmlns:at="http://aterm.org">
  <at:value>1</at:value>
</at:int>

1

"abc"

<at:string xmlns:at="http://aterm.org">
  <at:value>abc</at:value>
</at:string>

"abc"

()

<at:tuple xmlns:at="http://aterm.org"/>

()

(1, 2)

<at:tuple xmlns:at="http://aterm.org">
  <at:int>
    <at:value>1</at:value>
  </at:int>
  <at:int>
    <at:value>2</at:value>
  </at:int>
</at:tuple>

(1,2)

[]

<at:list xmlns:at="http://aterm.org"/>

[]

[1, 2]

<at:list xmlns:at="http://aterm.org">
  <at:int>
    <at:value>1</at:value>
  </at:int>
  <at:int>
    <at:value>2</at:value>
  </at:int>
</at:list>

[1,2]

fred([foo, bar])

<at:appl xmlns:at="http://aterm.org" at:fun="fred">
  <at:list>
    <at:appl at:fun="foo"/>
    <at:appl at:fun="bar"/>
  </at:list>
</at:appl>

fred([foo,bar])

fred(None, [foo, bar])

<at:appl xmlns:at="http://aterm.org" at:fun="fred">
  <at:appl at:fun="None"/>
  <at:list>
    <at:appl at:fun="foo"/>
    <at:appl at:fun="bar"/>
  </at:list>
</at:appl>

fred(None,[foo,bar])

fred(Some(barney), [foo, bar])

<at:appl xmlns:at="http://aterm.org" at:fun="fred">
  <at:appl at:fun="Some">
    <at:appl at:fun="barney"/>
  </at:appl>
  <at:list>
    <at:appl at:fun="foo"/>
    <at:appl at:fun="bar"/>
  </at:list>
</at:appl>

fred(Some(barney),[foo,bar])

fred("foo", "bar")

<at:appl xmlns:at="http://aterm.org" at:fun="fred">
  <at:string>
    <at:value>foo</at:value>
  </at:string>
  <at:string>
    <at:value>bar</at:value>
  </at:string>
</at:appl>

fred("foo","bar")

foo{fred}

<at:appl xmlns:at="http://aterm.org" at:fun="foo">
  <at:anno>
    <at:appl at:fun="fred"/>
  </at:anno>
</at:appl>

foo{fred}

Implicit XML

foo

<foo xmlns:at="http://aterm.org"/>

foo

foo(1)

<foo xmlns:at="http://aterm.org">1</foo>

foo("1")

1

"abc"

()

(1, 2)

[]

[1, 2]

fred([foo, bar])

<fred xmlns:at="http://aterm.org">
  <foo/>
  <bar/>
</fred>

fred(foo,bar)

fred(None, [foo, bar])

<fred xmlns:at="http://aterm.org">
  <foo/>
  <bar/>
</fred>

fred(foo,bar)

fred(Some(barney), [foo, bar])

<fred xmlns:at="http://aterm.org">
  <barney/>
  <foo/>
  <bar/>
</fred>

fred(barney,foo,bar)

fred("foo", "bar")

<fred xmlns:at="http://aterm.org">foobar</fred>

fred("foobar")

foo{fred}

<foo xmlns:at="http://aterm.org">
  <at:anno>
    <fred/>
  </at:anno>
</foo>

foo{fred}

Tools

• term-to-dot -- transforms an ATerm to a graph in the DotLanguage
• pp-aterm -- pretty prints an ATerm to text with whitespace to show the tree structure of the ATerm.

box2asfix

Synopsis

box2asfix [-i box-term] [-o asfix-tree] [-w width] -t asfix-tree

Description

The utility box2asfix produces an AsFix parse-tree in which the layout nodes are updated according to the formatting specified in `box-tree'.

The utility takes as input an asfix-term (specified with the `-t' option) in which the layout nodes needs to be updated, and a box-term (specified with the `-o' option) which describes how the layout nodes should be updated.

The -w option can be used to specify the maximal line width.The line width defaults to 80 columns.

Options

-h : display usage information. Use this option to get information on additional options

-q : run quietly

-v : display version information

Limitations

The box2asfix utility only accepts AsFix1^? parse-trees.

See also

GPP, HowToPrettyPrintAGrammar

abox2html

Synopsis

abox2html [-c] [-i box-term] [-o html-file] [-t title]

Description

The utility abox2html produces an HTML file according to the formatting defined in box-term.

Options

-c

Generate cascading style sheet. The HTML code requirers a cascading style sheet (named `box2html.css') which contains specific formatting information. An initial style-sheet can be obtained by specifying the `-c' option to box2html. This style-sheet can be modified manually to customize the appearance of the HTML code in your web-browser.

-h

display usage information. Use this option to get information on additional options

-t <title>

Specify title of html page

-v

display version information

See also

GenericPrettyPrinter, AstToAbox^?, HowToPrettyPrintAGrammar

abox2latex

Synopsis

abox2latex [--alltt] [--boxenv] [-i box-term] [-o html-file] [-w width] [-t abbreviations_file] Description

The utility abox2latex produces LaTeX^? code according to the formatting defined in box-term. See HowToUseGPPWithLaTeX for instructions about how to use the generated LaTeX^? code in your LaTeX^? documents.

Options

--alltt

Instructs abox2latex to produce an alltt environment. See HowToUseGPPWithLaTeX how this serves layout-preserving pretty-printing

--boxenv

Instructs abox2latex to produce a fully formatted boxenv environment (default)

-h

display usage information. Use this option to get information on additional options

-t <table>

With this switch latex abreviations can be specified. The abox2latex utility can replace ordinary BOX strings by LaTeX^? commands. These mappings can be defined in abbreviation tables. For example, to specify that the string "\\//" should be replaced by abox2latex to the LaTeX^? command \vee, the abbreviations table should look like:

[
 ["\\/", "\\ensuremath{\\vee}"]
]

-v

display version information

-w <width>

Table containing box to latex macro mappings. With this switch the maximum line width can be specified. The default line width equals 80 columns.

See also

GenericPrettyPrinter, AstToAbox^?, AsFixToAbox, HowToPrettyPrintAGrammar, HowToUseGPPWithLaTeX

An abstract composition definition is a definition of a composition that is independent of the type of components. It describes the components in the composition, the relation between components, the structure of the composition, and it contains URLs to locations where different component types are stored.

Types of components can be build-level components, source files, petri nets, etc.

Example

Below is an example of an abstract composition definition for the composition described at KoalaCompiler.

component koala-bundle (BUNDLE)
{
   path = koala-bundle
}
{
}

component asfix-tools (ASFIX-TOOLS)
{
   path = koala-bundle/asfix-tools
   url = file:///home/mdejonge/pkgs
}
{
   IbinASFIX-TOOLS : asfix-tools -> koala-bundle/asfix-tools/asfix-tools-1_0
   IlibATERM : aterm -> koala-bundle/aterm
   IlibXTC : xtc -> koala-bundle/stratego
   IlibSRTS : srts -> koala-bundle/stratego
}

module asfix-tools-1_0
{
   path = koala-bundle/asfix-tools/asfix-tools-1_0
   url = file:///home/mdejonge/pkgs
}
{
   IlibATERM : aterm -> koala-bundle/asfix-tools
   IlibXTC : xtc -> koala-bundle/asfix-tools
   IlibSRTS : srts -> koala-bundle/asfix-tools
}

component aterm (ATerm)
{
   path = koala-bundle/aterm
   url = file:///home/mdejonge/pkgs
}
{
   IlibATERM : aterm -> koala-bundle/aterm/aterm-1_0
}

module aterm-1_0
{
   path = koala-bundle/aterm/aterm-1_0
   url = file:///home/mdejonge/pkgs
}
{
}

component gpp (GPP)
{
   path = koala-bundle/gpp
   url = file:///home/mdejonge/pkgs
}
{
   IbinGPP : gpp -> koala-bundle/gpp/gpp-1_0
   IlibATERM : aterm -> koala-bundle/aterm
   IbinASFIX-TOOLS : asfix-tools -> koala-bundle/asfix-tools
   IbinSGLR : sglr -> koala-bundle/sglr
   IbinGRAPH-TOOLS : graph-tools -> koala-bundle/graph-tools
   IlibSRTS : srts -> koala-bundle/stratego
   IlibXTC : xtc -> koala-bundle/stratego
}

module gpp-1_0
{
   path = koala-bundle/gpp/gpp-1_0
   url = file:///home/mdejonge/pkgs
}
{
   IlibATERM : aterm -> koala-bundle/gpp
   IbinASFIX-TOOLS : asfix-tools -> koala-bundle/gpp
   IbinSGLR : sglr -> koala-bundle/gpp
   IbinGRAPH-TOOLS : graph-tools -> koala-bundle/gpp
   IlibSRTS : srts -> koala-bundle/gpp
   IlibXTC : xtc -> koala-bundle/gpp
}

component graph-tools (GRAPH-TOOLS)
{
   path = koala-bundle/graph-tools
   url = file:///home/mdejonge/pkgs
}
{
   IbinGRAPH-TOOLS : graph-tools -> koala-bundle/graph-tools/graph-tools-1_0
   IlibATERM : aterm -> koala-bundle/aterm
   IlibSRTS : srts -> koala-bundle/stratego
   IlibXTC : xtc -> koala-bundle/stratego
}

module graph-tools-1_0
{
   path = koala-bundle/graph-tools/graph-tools-1_0
   url = file:///home/mdejonge/pkgs
}
{
   IlibATERM : aterm -> koala-bundle/graph-tools
   IlibXTC : xtc -> koala-bundle/graph-tools
   IlibSRTS : srts -> koala-bundle/graph-tools
}

component sglr (mySGLR)
{
   path = koala-bundle/sglr
   url = file:///home/mdejonge/pkgs
}
{
   IbinSGLR : sglr -> koala-bundle/sglr/sglr-1_0
   IlibPT-SUPPORT : pt-support -> koala-bundle/sglr/pt-support-1_0
   IlibTOOLBUS-LIB : toolbus-lib -> koala-bundle/sglr/toolbus-lib-1_0
   IlibATERM : aterm -> koala-bundle/aterm
}

module sglr-1_0
{
   path = koala-bundle/sglr/sglr-1_0
   url = file:///home/mdejonge/pkgs
}
{
   IlibPT-SUPPORT : pt-support -> koala-bundle/sglr
   IlibTOOLBUS-LIB : toolbus-lib -> koala-bundle/sglr
   IlibATERM : aterm -> koala-bundle/sglr
}

module pt-support-1_0
{
   path = koala-bundle/sglr/pt-support-1_0
   url = file:///home/mdejonge/pkgs
}
{
   IlibTOOLBUS-LIB : toolbus-lib -> koala-bundle/sglr
   IlibATERM : aterm -> koala-bundle/sglr
}

module toolbus-lib-1_0
{
   path = koala-bundle/sglr/toolbus-lib-1_0
   url = file:///home/mdejonge/pkgs
}
{
   IlibATERM : aterm -> koala-bundle/sglr
}

component stratego (Stratego)
{
   path = koala-bundle/stratego
}
{
   IlibSRTS : srts -> koala-bundle/stratego/my_srts
   IlibXTC : xtc -> koala-bundle/stratego/my_xtc
   IlibATERM : aterm -> koala-bundle/aterm
}

component my_xtc (XTC)
{
   path = koala-bundle/stratego/my_xtc
   url = file:///home/mdejonge/pkgs
}
{
   IlibXTC : xtc -> koala-bundle/stratego/my_xtc/xtc-1_0
   IlibATERM : aterm -> koala-bundle/stratego
   IlibSRTS : srts -> koala-bundle/stratego
}

module xtc-1_0
{
   path = koala-bundle/stratego/my_xtc/xtc-1_0
   url = file:///home/mdejonge/pkgs
}
{
   IlibATERM : aterm -> koala-bundle/stratego/my_xtc
   IlibSRTS : srts -> koala-bundle/stratego/my_xtc
}

component my_srts (SRTS)
{
   path = koala-bundle/stratego/my_srts
   url = file:///home/mdejonge/pkgs
}
{
   IlibSRTS : srts -> koala-bundle/stratego/my_srts/srts-1_0
   IlibATERM : aterm -> koala-bundle/stratego
}

module srts-1_0
{
   path = koala-bundle/stratego/my_srts/srts-1_0
   url = file:///home/mdejonge/pkgs
}
{
   IlibATERM : aterm -> koala-bundle/stratego/my_srts
}

component koala (KOALA)
{
   path = koala-bundle/koala
   url = file:///home/mdejonge/pkgs
}
{
   IbinKOALA : koala -> koala-bundle/koala/koala-1_0
   IbinGPP : gpp -> koala-bundle/gpp
   IbinSGLR : sglr -> koala-bundle/sglr
   IlibSRTS : srts -> koala-bundle/stratego
   IlibXTC : xtc -> koala-bundle/stratego
   IlibATERM : aterm -> koala-bundle/aterm
}

module koala-1_0
{
   path = koala-bundle/koala/koala-1_0
   url = file:///home/mdejonge/pkgs
}
{
   IbinGPP : gpp -> koala-bundle/koala
   IbinSGLR : sglr -> koala-bundle/koala
   IlibSRTS : srts -> koala-bundle/koala
   IlibXTC : xtc -> koala-bundle/koala
   IlibATERM : aterm -> koala-bundle/koala
}

-- MerijnDeJonge - 18 Feb 2004

Summary

addPosInfo adds position information to an AsFix2ME parse tree.

Example

The following syntax definition defines a tiny language of assignments and expressions.

  definition
  module Main
  exports
    context-free start-symbols Stm*

    sorts Id IntConst
    lexical syntax
      [\ \t\n]  -> LAYOUT
      [a-zA-Z]+ -> Id
      [0-9]+    -> IntConst

    sorts Stm
    context-free syntax
      Id ":=" Exp -> Stm {cons("Assign")}

    sorts Exp
    context-free syntax
      Id        -> Exp {cons("Var")}
      IntConst  -> Exp {cons("Int")}

      Exp "+"  Exp -> Exp  {left, cons("Plus")}

Generate a parse table:

  > sdf2table -i Foo.def -o Foo.tbl

Input file:

  x := 5
  y := x + 4
  z := x + y

Parse the input and have a look at the result in abstract syntax:

  > sglr -i foo.txt -p Foo.tbl | addPosInfo -p ./foo.txt -m | implodePT | pp-aterm

Notes:

• you must specify the path (-p) of the input file.
• sglr must produce Asfix2ME. This happens if you pass no -2 argument.

Name

ambtracker -- display the productions in a parse tree causing ambiguities

Description

Compared to visamb ambtracker offers an alternative visualization of ambiguities in a parse tree. ambtracker just displays the productions causing the ambiguity and not how these productions are applied to the actual input. The location (row and column) in the input where an ambiguity is caused is also displayed for easy reference.

Consider the syntax definition that is used to illustrate visamb.

  definition
  module Main
  exports
    sorts Exp
    lexical syntax
      [\ \t\n] -> LAYOUT
    context-free syntax
      "id"    -> Exp
      Exp Exp -> Exp

From this syntax definition an SGLR parse table can be generated:

  sdf2table -i Exp.sdf -o Exp.tbl

The ambiguities of the phrase id id id can be shown with:

  echo "id id id" | sglr -2 -p Exp.tbl | ambtracker

the output of this command is:

  1 ambiguity cluster:

  [1/1] at (1:0):
    Exp  Exp -> Exp
    Exp  Exp -> Exp

Description

AsFix (ASF+SDF fixed format) is a format for representing parse trees in the ATerm format.

Currently two versions of AsFix are in used: AsFix2ME and AsFix2. AsFix2 is extremely verbose since layout and literals have a structured representation in AsFix2. AsFix2ME is a more concise since lists, layout and literals are flattened. This is the parse tree format that is used in the ASF+SDF Meta-Environment.

Application

The SGLR parser outputs the result of parsing an input in the AsFix format. The most recent versions of SGLR produce AsFix2ME by default. SGLR produces AsFix2 when the -2 flag is used.

In StrategoXT program transformation systems usually operate on abstract syntax trees. The AsFix2 output of SGLR is transformed into an abstract syntax tree by implode-asfix.

Format Internals

Usually you don't actually want to see an AsFix2 or AsFix2ME parse tree. AsFix takes the job of representing the result of parsing an input very serious. It includes all information required to reproduce the original input. This unparsing is implemented by the asfix-yield tool. The AsFix format exactly describes what kind of productions are applied. An AsFix representation is self-contained: all grammar information needed to interpret a term is also included.

Example

Consider the following SDF syntax definition.

  definition
  module Exp
  exports
    sorts Exp

    lexical syntax
      [\ \t\n]  -> LAYOUT
      [a-zA-Z]+ -> Id
      [0-9]+    -> IntConst

    context-free syntax
      Id        -> Exp {cons("Var")}
      IntConst  -> Exp {cons("Int")}

      Exp "*"  Exp -> Exp  {left, cons("Mul")}
      Exp "/"  Exp -> Exp  {left, cons("Div")}
      Exp "%"  Exp -> Exp  {left, cons("Mod")}

      Exp "+"  Exp -> Exp  {left, cons("Plus")}
      Exp "-"  Exp -> Exp  {left, cons("Minus")}

    context-free priorities
      {left:
        Exp "*"  Exp -> Exp
        Exp "/"  Exp -> Exp
        Exp "%"  Exp -> Exp
      }
    > {left:
        Exp "+"  Exp -> Exp
        Exp "-"  Exp -> Exp
      }

PGEN's sdf2table produces an SGLR parse table from this syntax definition:

  sdf2table -m Exp -i Exp.def -o Exp.tbl

Parsing to AsFix2ME

Parsing the expression 1 + a to the compact AsFix variant AsFix2ME using:

  echo "1 + a" | sglr -m -A -p Exp.tbl | pp-aterm

produces the following parse tree:

parsetree(
  appl(
    prod(
      [cf(opt(layout)), cf(sort("Exp")), cf(opt(layout))]
    , sort("<START>")
    , no-attrs
    )
  , [ appl(prod([], cf(opt(layout)), no-attrs), [])
    , appl(
        prod(
          [ cf(sort("Exp"))
          , cf(opt(layout))
          , lit("+")
          , cf(opt(layout))
          , cf(sort("Exp"))
          ]
        , cf(sort("Exp"))
        , attrs([assoc(left), term(cons("Plus"))])
        )
      , [ appl(
            prod(
              [cf(sort("IntConst"))]
            , cf(sort("Exp"))
            , attrs([term(cons("Int"))])
            )
          , [ appl(
                prod(
                  [lex(sort("IntConst"))]
                , cf(sort("IntConst"))
                , no-attrs
                )
              , [ appl(
                    list(iter-star(char-class([range(0, 255)])))
                  , [49]
                  )
                ]
              )
            ]
          )
        , appl(
            prod([cf(layout)], cf(opt(layout)), no-attrs)
          , [32]
          )
        , lit("+")
        , appl(
            prod([cf(layout)], cf(opt(layout)), no-attrs)
          , [32]
          )
        , appl(
            prod(
              [cf(sort("Id"))]
            , cf(sort("Exp"))
            , attrs([term(cons("Var"))])
            )
          , [ appl(
                prod(
                  [lex(sort("Id"))]
                , cf(sort("Id"))
                , no-attrs
                )
              , [ appl(
                    list(iter-star(char-class([range(0, 255)])))
                  , [97]
                  )
                ]
              )
            ]
          )
        ]
      )
    , appl(
        prod([cf(layout)], cf(opt(layout)), no-attrs)
      , [10]
      )
    ]
  )
, 0
)

Parsing to AsFix2

Parsing the same expression to AsFix2:

  echo "1 + a" | sglr -2 -A -p Exp.tbl | pp-aterm

results in:

parsetree(
  appl(
    prod(
      [cf(opt(layout)), cf(sort("Exp")), cf(opt(layout))]
    , sort("<START>")
    , no-attrs
    )
  , [ appl(prod([], cf(opt(layout)), no-attrs), [])
    , appl(
        prod(
          [ cf(sort("Exp"))
          , cf(opt(layout))
          , lit("+")
          , cf(opt(layout))
          , cf(sort("Exp"))
          ]
        , cf(sort("Exp"))
        , attrs([assoc(left), term(cons("Plus"))])
        )
      , [ appl(
            prod(
              [cf(sort("IntConst"))]
            , cf(sort("Exp"))
            , attrs([term(cons("Int"))])
            )
          , [ appl(
                prod(
                  [lex(sort("IntConst"))]
                , cf(sort("IntConst"))
                , no-attrs
                )
              , [ appl(
                    prod(
                      [lex(iter(char-class([range(48, 57)])))]
                    , lex(sort("IntConst"))
                    , no-attrs
                    )
                  , [ appl(
                        prod(
                          [char-class([range(48, 57)])]
                        , lex(iter(char-class([range(48, 57)])))
                        , no-attrs
                        )
                      , [49]
                      )
                    ]
                  )
                ]
              )
            ]
          )
        , appl(
            prod([cf(layout)], cf(opt(layout)), no-attrs)
          , [ appl(
                prod([lex(layout)], cf(layout), no-attrs)
              , [ appl(
                    prod(
                      [char-class([range(9, 10), 32])]
                    , lex(layout)
                    , no-attrs
                    )
                  , [32]
                  )
                ]
              )
            ]
          )
        , appl(
            prod([char-class([43])], lit("+"), no-attrs)
          , [43]
          )
        , appl(
            prod([cf(layout)], cf(opt(layout)), no-attrs)
          , [ appl(
                prod([lex(layout)], cf(layout), no-attrs)
              , [ appl(
                    prod(
                      [char-class([range(9, 10), 32])]
                    , lex(layout)
                    , no-attrs
                    )
                  , [32]
                  )
                ]
              )
            ]
          )
        , appl(
            prod(
              [cf(sort("Id"))]
            , cf(sort("Exp"))
            , attrs([term(cons("Var"))])
            )
          , [ appl(
                prod(
                  [lex(sort("Id"))]
                , cf(sort("Id"))
                , no-attrs
                )
              , [ appl(
                    prod(
                      [ lex(
                          iter(
                            char-class([range(65, 90), range(97, 122)])
                          )
                        )
                      ]
                    , lex(sort("Id"))
                    , no-attrs
                    )
                  , [ appl(
                        prod(
                          [char-class([range(65, 90), range(97, 122)])]
                        , lex(
                            iter(
                              char-class([range(65, 90), range(97, 122)])
                            )
                          )
                        , no-attrs
                        )
                      , [97]
                      )
                    ]
                  )
                ]
              )
            ]
          )
        ]
      )
    , appl(
        prod([cf(layout)], cf(opt(layout)), no-attrs)
      , [ appl(
            prod([lex(layout)], cf(layout), no-attrs)
          , [ appl(
                prod(
                  [char-class([range(9, 10), 32])]
                , lex(layout)
                , no-attrs
                )
              , [10]
              )
            ]
          )
        ]
      )
    ]
  )
, 0
)

Producing an Abstract Syntax Tree

Applying implode-asfix to reduce this to an abstract syntax tree:

  echo "1 + a" | sglr -2A -p Exp.tbl | implode-asfix | pp-aterm

  Plus(Int("1"), Var("a"))

implode-asfix only accepts AsFix2. The implodePT (part of the pt-support package, which is in the sdf2-bundle) implements the same implosion for AsFix2ME.

  echo "1 + a" | sglr -mA -p Exp.tbl | implodePT | pp-aterm

  Plus(Int("1"), Var("a"))

Literals in AsFix2ME

Consider the syntax definition

  definition
  module Exp
  exports
    sorts Exp
    context-free syntax
      "nil" -> Exp

    lexical syntax
      [\t\n\r\ ] -> LAYOUT

Representation in AsFix2ME:

  appl(
    prod([lit("nil")], cf(sort("Exp")), no-attrs)
  , [lit("nil")]
  )

Representation in AsFix2:

  appl(
    prod([lit("nil")], cf(sort("Exp")), no-attrs)
  , [ appl(
        prod(
          [char-class([110]), char-class([105]), char-class([108])]
        , lit("nil")
        , no-attrs
        )
      , [110, 105, 108]
      )
    ]
  )

Layout in AsFix2ME

Lexical syntax in AsFix2ME

Consider the syntax definition

  definition
  module Exp
  hiddens
    sorts IntConst

  exports
    sorts Exp
  
    lexical syntax
      [\ \t\n]  -> LAYOUT
      [0-9]+    -> IntConst
  
    context-free syntax
      IntConst  -> Exp {cons("Int")}

  sdf2table -i Exp-tiny.def.def -o Exp-tiny.tbl -m Exp

Stratego Script to extract the tree related to the context-free symbol IntConst:

  import simple-traversal ;;
  oncetd(?appl(prod(_, cf(sort("IntConst")), _), _); ?x) ;;
  !x ;;

Parse the expression "12" to AsFix2^?

echo "1" | sglr -2 -p Exp-tiny.tbl | stratego-shell --script extract-lex.srts | pp-aterm

appl(
  prod(
    [lex(sort("IntConst"))]
  , cf(sort("IntConst"))
  , no-attrs
  )
, [ appl(
      prod(
        [lex(iter(char-class([range(48, 57)])))]
      , lex(sort("IntConst"))
      , no-attrs
      )
    , [ appl(
          prod(
            [ lex(iter(char-class([range(48, 57)])))
            , lex(iter(char-class([range(48, 57)])))
            ]
          , lex(iter(char-class([range(48, 57)])))
          , attrs([assoc(left)])
          )
        , [ appl(
              prod(
                [char-class([range(48, 57)])]
              , lex(iter(char-class([range(48, 57)])))
              , no-attrs
              )
            , [49]
            )
          , appl(
              prod(
                [char-class([range(48, 57)])]
              , lex(iter(char-class([range(48, 57)])))
              , no-attrs
              )
            , [50]
            )
          ]
        )
      ]
    )
  ]
)

  echo "12" | sglr -m -p Exp-tiny.tbl | stratego-shell --script extract-cf.strs | pp-aterm

appl(
  prod(
    [lex(sort("IntConst"))]
  , cf(sort("IntConst"))
  , no-attrs
  )
, [ appl(
      list(iter-star(char-class([range(0, 255)])))
    , [49, 50]
    )
  ]
)

• Note: If we add a constructor to the IntConst production the constructor name will be lost.

Now consider the syntax definition:

  definition
  module Exp
  hiddens
    sorts DecConst

  exports
    sorts Exp
  
    lexical syntax
      [\ \t\n]  -> LAYOUT
      [0-9]+ "." [0-9]+ -> DecConst {cons("DecConst")}
  
    context-free syntax
      DecConst  -> Exp

Parsing to AsFix2ME:

  echo "13.25" | sglr -m -p Exp-tiny.tbl | stratego-shell --script extract-cf.strs | pp-aterm

appl(
  prod(
    [lex(sort("DecConst"))]
  , cf(sort("DecConst"))
  , no-attrs
  )
, [ appl(
      list(iter-star(char-class([range(0, 255)])))
    , [49, 51, 46, 50, 53]
    )
  ]
)

Syntax in AsFix2ME

Consider the syntax definition

  definition
  module Exp
  hiddens
    sorts DecConst

  exports
    sorts Exp
    context-free syntax
      DecConst  -> Exp

    syntax
      [0-9]+ "." [0-9]+ -> DecConst {cons("DecConst")}
      DecConst -> <DecConst-CF>

    lexical syntax
      [\t\n\r\ ] -> LAYOUT

In this case the DecConst application is represented as a tree in AsFix2ME, just like in AsFix2.

Description

An asfix to asfix tool that reserves comments that were of the input source code by putting them in annotations of the AST.

It is difficult to decide what a comment actually comments on. The tool uses the heuristic that a comment is usually about the next construct. Therefore, it adds the comment to the first term with a constructor in the the subtree of the next symbol in the production rule. If this next symbol is a literal, then the comment will not be preserved.

Example

This simple tool works remarkably well. Let me illustrate this with an example input that I used during the development of this tool.

  /**
   * Voodoo
   */
  class Voodoo
  {
    /**
     * Bla bla
     */
    public void foo(/*let me explain this */ int x)
    {
      // just return
      return /* foo */ x;
    }
  }

An example fragment of the AST:

  Param([], Int, Id("x")){(Comment, "/*let me explain this */")}

The JavaFront pretty-printer has been extended to support these Comment annotations. The following pipe:

  sglr -2 -s CompilationUnit -p ~/wc/java-front/syn/v1.5/Java-15.tbl 
     -i ~/Foo.java
  | ./asfix-anno-comments
  | implode-asfix
  | pp-java

produces the following output:

  /**
   * Voodoo
   */
  class Voodoo
  {
    /**
     * Bla bla
     */
    public void foo(/*let me explain this */ int x)
    {
      // just return
      return /* foo */ x;
    }
  }

asfix2abox

Synopsis

asfix2abox [-c] [-i asfix-tree] [-o box-term] [-p table1] [-p table2] ...

Description

The asfix2abox utility is a generic formatter that maps a parse-tree represented in AsFix to BOX according to the pretty-print rules specified in the pretty-print tables <table1> <table2> ...

The utility accepts an ordered sequence of pretty-print tables which define mappings from language constructs to BOX expressions (See HowToDefinePrettyPrintTables). The tables are ordered such that the pretty-print rules in <table1> have higher precedence than the entries in <table2>.

The BOX term that is produced by asfix2abox can be passed to one of the available back-ends to obtain a plain text, HTML, or LaTeX representation of your input term (See HowToPrettyPrintAGrammar).

Options

-c

Conservative pretty-printing (only format where needed).

-h

display usage information. Use this option to get information on additional options

-p <table>

Use pretty-print rules defined in <table>. Multiple tables can be specified

-v

display version information

See also

GenericPrettyPrinter, HowToPrettyPrintAGrammar, HowToDefinePrettyPrintTables

Tools

• implode-asfix -- implodes a parse tree in AsFix2 format to an abstract syntax tree
• asfix-yield -- unparses an AsFix2 parse tree to flat text.
• visamb -- display the ambiguities in a parse tree represented in AsFix2
• asfix-anno-comments -- put comments in annotations of a parse tree

Compiling and installing software packages is often a time consuming and complicated business. You have to read the installation instructions to determine which steps are required for building and installing the software, then you have to determine how to configure the system (indicate where components such as libraries can be found and which compile flags should be used). Finally, you have to execute configure, compile, and install tasks manually. When you have to install the same software on multiple architectures, you might come in the situation where you have to perform these steps for each platform again. To make things even worse, this process has to be repeated each time you want to install a new version of the software package.

The "autobuild" software package tries to solve this problem by introducing the notion of so called "build files" in which you can define how a particular software system should be built on different platforms. The "build" tool reads a build file, and builds the software package for you on each platform you defined. It can perform the build processes on multiple platforms in parallel, and it creates informative HTML and textual log files.

Two typical applications of autobuild are:

1. Centralized management of software packages for multiple architectures. For instance, part of the software installed at the Transform.SEN1 research group of the CWI is controlled by build files. Once a new version of a package needs to be installed (or an installation needs to be performed again), a single call to build suffices. All build files are under CVS control to keep track of changes made to the files over time.
2. Daily builds where the build tool is used to compile software under development on a daily basis. The generated HTML documentation gives access to the build log and helps locating errors in the daily build. (See DailyBuildSystem).

Getting the Software

The autobuild package is open source and distributed as a gzipped tar file named `autobuild-<version>.tar.gz', where <version> denotes the version of the package. You can download the package at:

• http://www.cs.uu.nl/~mdejonge/downloads/autobuild-0.5.tar.gz

• Is it possible to have localhost as a platform? How do I indicate that in the platforms file?

You can simply add an antry like ' myplatform="localhost" ' in your platforms file.

• The README file has information about setting up the BuildFile^? for one package, but does not explain how to set up a complete dailybuild environment. Would it be possible to provide a package with a template or a sample set-up?

AutoBuild is not a daily-build system by itself. Checkout DailyBuildSystem for setting up a daily-build system with the dbs package.

At this moment you cannot use a template set-up. You can however use the '-g' switch to generate an initial Buildfile for a package and use AutoBuild's `include' mechanism to import standard (shared) build configuration. See the Buildfiles at http://www.cwi.nl/~daybuild/ for examples.

Autobundle is a utility for making software distributions by bundling multiple (third-party) software packages. Autobundle promotes the development of reusable components which can be distributed separately, or as part of larger software systems. Autobundle thus allows you to easily distribute your software as self-contained packages (by bundling all required components). This greatly simplifies the installation process of your software because a user does not need to obtain, configure, and install these required software packages first.

Two typical applications of autobundle are:

1. The distribution of the ASF+SDF Meta-Environment (www.cwi.nl/projects/MetaEnv), which bundles amongst others the ToolBus^? software application architecture, the ATerms abstract data type and libraries, a parser, and a parser generator. All these components are also distributed and used separately.
2. XT, which stands for `Program Transformations Tools' (www.program-transformation.org/xt). It bundles some of the components of the ASF+SDF Meta-Environment (the ATerms abstract data type, the parser, and parser generator), but in addition the programming language Stratego (www.stratego-language.org), a pretty-printer (www.cs.uu.nl/~mdejonge/gpp),the grammar tools (GT) package (www.program-transformation.org/gt), and the grammar bases (GB) package (www.program-transformation.org/gb).

Autobundle allows you to easily define such distributions by listing the components, together with a version number, and the location where the components can be retrieved. A typical autobundle distribution is thus simply a collection of components of a particular version.

The autobundle software helps creating an initial framework for the package and creates the software bundle for you, by downloading the different components and building a single distribution from them. Autobundle requires that the components are constructed using AutoConf^? and AutoMake^?. The use of AutoConf^? and AutoMake^? is clearly described in the documentation of both packages and not addressed here.

Available packages are stored in the Online Package Base, from where you can obtain generated software bundles by making specific packages selections.

Publications

• Source Tree Composition

• Package-Based Software Development

• Feature-Based Product Line Instantiation using Source-Level Packages

• The Linux Kernel as Flexible Product-Line Architecture

Getting the software

The autobundle package is open source and the most recent release can be obtained from:

• http://nix.cs.uu.nl/dist/stratego/autobundle-0.14/

Documentation

• How to pretty print a grammar
• How to define pretty print tables

Publications

• http://www.cs.uu.nl/groups/ST/twiki/bin/view/Merijn/PaperPrettyPrintingForSoftwareReengineering
• http://www.cs.uu.nl/groups/ST/twiki/bin/view/Merijn/PaperPrettyPrinterForEveryOccasion
• http://www.cs.uu.nl/groups/ST/twiki/bin/view/Merijn/PaperALaTeXStyleFileForFormattingBOXExpressions
• http://www.cs.uu.nl/groups/ST/twiki/bin/view/Merijn/PaperToReuseOrToBeReused

Description

The `dbs' package implements an open framework for daily-builds (i.e. regular performed builds to continuously verify the consistence of all parts of a software system). The dbs package is based on AutoBuild and performs the builds for all software package that you put under control of dbs. Furthermore, it generates web pages with summaries of the build processes. The package supports software construction on multiple architectures.

As an example of an operational daily-build system you might want to take a look at:

• http://www.cwi.nl/~daybuild (the dailybuild system at the CWI)

Download

The dbs package is open source and distributed as a gzipped tar file named `dbs-<version>.tar.gz', where <version> denotes the version of the package. You can download the package at:

• http://www.cs.uu.nl/~mdejonge/downloads/dbs-0.3.tar.gz

Discussion

How would this compare to the Cruise Control Continuous Integration Toolkit? See http://cruisecontrol.sourceforge.net/

-- ArieVanDeursen 30 Okt 2001.

-----------------------------------------------------------------------------

RECOVERY OF SYNTAX DEFINITION FOR LEX

-----------------------------------------------------------------------------

No syntax definition for LEX was available in the grammar-base. In
order to further automate the translation of LEX/YACC grammars to SDF2
syntax definitions, such a syntax definition is needed. In this file I
report the steps I took to get LEX in SDF2.

-- Eelco Visser 2001/09/29

-----------------------------------------------------------------------------

[Step 1] Locate the sources

   ftp://ftp.gnu.org/non-gnu/flex/

[Step 2] Inspect the source

   cd flex-2.5.4
   less parse.y 

[Step 3] Copy source to grammar base

   cp parse.y ~/res/XT/gb/grammars/lex.0

[Step 3] Parse the YACC (Bison) source

>    parse -l yacc -i lex.y -I -o lex.af 

=>    Error: charliteral '\n' not recognized. Repair syntax definition of
   YACC.

[Step 4] Translate to AbstractSDF

>   parse -l yacc -i lex.y -I -o lex.af 
   yacc2sdf -i lex.af -o lex.asdf 

[Step 5] Pretty-print syntax definition and inspect

>   sdf-bracket -i lex.asdf | pp -a -l sdf -o lex.def -v 2.1 
   less lex.def    

[Step 6] Regularize the syntax definition

>   parse -l yacc -i lex.y -I -o lex.af 
   yacc2sdf -i lex.af -o lex.asdf 
   sdf-regularize -i lex.asdf -o lex.reg.asdf 
   sdf-bracket -i lex.reg.asdf | pp -a -l sdf -o lex.def -v 2.1 
   less lex.def    

[Step 7] Generate constructors

>   parse -l yacc -i lex.y -I -o lex.af 
   yacc2sdf -i lex.af -o lex.asdf 
   sdf-regularize -i lex.asdf -o lex.reg.asdf 
   sdf-cons -i lex.reg.asdf -o lex.reg.cons.asdf
   sdf-bracket -i lex.reg.cons.asdf | pp -a -l sdf -o lex.def -v 2.1 
   less lex.def   

[Step 8] Edit the definition to define lexical syntax and improve constructors

[Step 9] Unpack the definition to create separate SDF modules. Make check can then
   be used to generate lex.def and automatically parse various example files.
   Before doing this change the names of the modules Lexical and Generated into
     Lex-Symbols and Lex, respectively.

>   unpack-sdf lex.def

[Step 10] Further edit the modules to define lexical syntax.

=>    It turns rather hard to parse complete lex files. The lexical syntax
   is very tricky. Instead I decide to reduce the problem by editing lex
   files by hand to remove all irrelevant stuff and only leave definitions
   of the form |name re| and rules of the form |re return id;|. Newlines
   cannot be used as general layout, but are used to delimit definitions
   and rules. No superfluous newlines are allowed.

=>    Succeed in parsing stratego.mod.l, a modified version of the stratego lexical
   syntax in lex.

[Step 11] Improve the syntax definition to get good abstract syntax. Start with
   unfolding literals.

>   parse -l sdf -v 2.1 -I -i lex.def -o lex.adef
   unfold-literal -i lex.adef -o lex.unf.adef
   sdf-bracket -i lex.adef | pp -a -l sdf -o lex.def -v 2.1 
   less lex.def   

=>    Automatic application does not work, do it manually. (come back and
   repair unfold-literal later)

[Step 12] Abstract syntax looks good. Install parse table such that it can be used with the
   parse tool of the grammar base.

>    make install
   parse -l lex -i data/stratego.mod.l -I


[Step 13] Project finished. 

=>    Future work: 
   - parse full lex definitions
   - generate a signature from the syntax definition

----------------------------------------------------------------------

RECOVERING A SYNTAX DEFINITION FOR STRATEGO

----------------------------------------------------------------------

This directory contains a syntax definition in SDF2 of the Stratego
language. This file describes step by step how the syntax definition
was obtained from the YACC source in the source tree of the Stratego
Compiler SC.  Including the search for tools, their usage, repair if
necessary and the occasional implementation of a missing tool.

-- Eelco Visser 1/10/2001

----------------------------------------------------------------------

[step 1] Copy the YACC file

>   cp ../../sc/spec/syn/stratego.grm .   

[step 2] Parse the YACC file

>   parse -l yacc -i stratego.grm -I -o stratego.af 

[step 3] Translate YACC to SDF

>   yacc2sdf -i stratego.af -o stratego.asdf   
  
[step 4] Pretty-print the syntax definition

>   pp -l sdf -i stratego.asdf -o stratego.def 

[step 4a] Find out what is wrong

>   pp -h  

=>    -a switch should be used to indicate that input is 
   abstract syntax

[step 4c] Pretty-print the syntax definition with -a 

>   pp -a -l sdf -i stratego.asdf -o stratego.def
   No pp entry found for: ["Definition"]
   rewriting failed 

=>    wrong pretty-print table              

[step 4d] Pretty-print the syntax definition with -a and sdf version 2.1

   pp -a -l sdf -i stratego.asdf -o stratego.def -v 2.1 

[step 5] Inspecting the generated syntax definition

>   less stratego.def   

=>    No templates for lexicals have been included. I remember that
   these were generated automatically from the %token declarations
   in the YACC file.
   
=>    Yes, yacc2sdf is broken; the signature of trees produced
   by the parse has been changed. Repair
   this.

=>    Problem was due to change in generation of constructors by
   sdf-cons. Adapted Yacc syntax definition; inlined injection
   Nmno* -> Nlist.

[Step 6] Repeat steps 1 - 5

>   parse -l yacc -i stratego.grm -I -o stratego.af 
   yacc2sdf -i stratego.af -o stratego.asdf 
   pp -a -l sdf -i stratego.asdf -o stratego.def -v 2.1  
   less stratego.def    

=>    Each template rule for token is put in its own lexical
   syntax section; merge these. Adapt yacc2sdf

[Step 7] Repeat step 6

>   parse -l yacc -i stratego.grm -I -o stratego.af 
   yacc2sdf -i stratego.af -o stratego.asdf 
   pp -a -l sdf -i stratego.asdf -o stratego.def -v 2.1  
   less stratego.def    

=>    Lexicals are now merged

=>    No list detection and transformation is done by
   yacc2sdf. Which tool does achieve that? 
    There used to be a deyaccification tool, which I can
   no longer find. 
    Write a new tool: sdf-regularize which achieves this.
   Add it to the sdf-tools package.

=>    sdf-regularize recognizes various kinds of lists and
   optional constructs and translates them into regular 
   expressions. The sorts representing these regular expressions
   are then superfluous. By inlining their definitions the
   sorts are removed and the syntax definition is shortened.

[Step 8] Add application of sdf-regularize and sdf-bracket

>   parse -l yacc -i stratego.grm -I -o stratego.af 
   yacc2sdf -i stratego.af -o stratego.asdf 
   sdf-regularize -i stratego.asdf -o stratego.reg.asdf 
   sdf-bracket -i stratego.reg.asdf       | pp -a -l sdf -o stratego.def -v 2.1  
   less stratego.def  

=>    The syntax definition looks good now, with much
   fewer productions. No definition for lexical syntax
   exists yet, however.

[Step 9] Generate constructor annotations

>   parse -l yacc -i stratego.grm -I -o stratego.af 
   yacc2sdf -i stratego.af -o stratego.asdf 
   sdf-regularize -i stratego.asdf -o stratego.reg.asdf 
   sdf-cons -i stratego.reg.asdf -o stratego.reg.cons.asdf
   sdf-bracket -i stratego.reg.cons.asdf       | pp -a -l sdf -o stratego.def -v 2.1 
   less stratego.def    

=>    This produces bad results, since the heuristics are
   based on the literals in the productions; we need
   to add in the literals. Can this be done automatically?

[Step 10] Find the LEX file

>   cp ../../sc/spec/syn/stratego.lx . 

=>    This looks like it could be translated into SDF2 mostly
   automatically.
   Let's find a Lex grammar. There is none in the grammar base.
   I guess we'll have to reverse engineer it.

=>    Created a syntax definition for a subset of LEX that can deal
   with stripped files that only contain definitions and rules,
   but not arbitrary C code. The file stratego.mod.l contains
   the stripped off lex file for stratego.

=>   The syntax definition for lex has been installed in the
   grammar base. We can now use the parse tool to parse stratego.mod.l

[Step 11] Parse the LEX file

>   parse -l lex -i stratego.mod.l -o stratego.mod.af -I  

[Step 12] Translating LEX to SDF2

=>    There is no tool for this yet. We'll have to write it.

=>    lex2sdf is new tool, implemented in grammar-recovery/src/yacc2sdf/

>   parse -l lex -i stratego.mod.l -o stratego.mod.af -I  
   lex2sdf -i stratego.mod.af -o stratego.mod.asdf 
   sdf-bracket -i stratego.mod.asdf       | pp -a -l sdf -o stratego.mod.sdf -v 2.1 
   less stratego.mod.sdf    

=>    This provides a good lexical syntax. Note that layout is missing.

[Step 13] Recapitulation. The following actions were taken to derive
   the SDF2 definition so far. Note that the source file names
   have been renamed

>   mv stratego.grm stratego.l
   mv stratego.mod.l stratego.l

=>    Context-free syntax

>   parse -l yacc -i stratego.y -I -o stratego-cfg.af 
   yacc2sdf -i stratego-cfg.af -o stratego-cfg.asdf 
   sdf-regularize -i stratego-cfg.asdf -o stratego-cfg.reg.asdf 
   sdf-bracket -i stratego-cfg.reg.asdf       | pp -a -l sdf -o stratego-cfg.def -v 2.1 
   less stratego-cfg.def

=>   Lexical syntax

>   parse -l lex -i stratego.l -o stratego-lex.af -I  
   lex2sdf -i stratego-lex.af -o stratego-lex.asdf 
   sdf-bracket -i stratego-lex.asdf       | pp -a -l sdf -o stratego-lex.def -v 2.1
   less stratego-lex.def

[Step 14] Combine lexical and context-free syntax

=>   edit: remove module Lexical from stratego-cfg.def

>    unpack-sdf stratego-cfg.def
    unpack-sdf stratego-lex.def

=>   edit: repair error in definition of Backslash

>    pack-sdf -i Main.sdf -I ./. -dep stratego.af

[Step 14] Unfold literals; replace token names by their definitions.

>    pack-sdf -i Main.sdf -I ./. -dep stratego.af -o stratego.af
   pp -A -l sdf -i stratego.af -o stratego.def -v 2.1
   
>   parse -l sdf -v 2.1 -i stratego.def -o stratego.af -I
   unfold-literal -i stratego.af -o stratego.unf.af
   sdf-bracket -i stratego.unf.af       | pp -a -l sdf -o stratego.unf.def -v 2.1
   less stratego.unf.def

=>    unfold-literals did not work for all cases. Rewrote the tool using
   dynamic rules, which made the specification much shorter.

=>    Use the result as the new Lexical.sdf and Stratego.sdf modules

>    unpack-sdf stratego.unf.def 

[Step 15] Clean up Lexical.sdf and Stratego.sdf manually

=>   Fill in missing lexical syntax
   - layout definitions (in particular definition of literate comments)

=>    Improve abstract syntax
   - unfold Optvarlist in StrategyDef
   - observations: often better to unfold optionals if the optional can
   also be expressed in terms of the 
   - example: the condition |where id| is equivalent to a rule without condition
   - example: the strategy definition |f = s| is equivalent to |f() = s|, i.e.,
     with an empty list of arguments
   - therefore the latter can be desugared into the former allowing uniform treatment
     without having to deal with None and Some constructors.
   - this is at the cost of extra productions, however, the *core* syntax is
     smaller

=>   Add constructors

>   make check

=>    Stratego files get parsed correctly it seems

[Step 16] Compatability with parse-mod. To use the parser based on the SDF2 definition
   it should be compatible with the existing YACC based parser.

=>    Set up a test script that checks compatability between SDF and YACC based parsers

=>   In order to achieve this abstract syntax trees should be desugared. 

=>    In order to implement a desugarer we need the signature of trees produced by
   the new parser.

[Step 17] Derive the signature of Stratego from the syntax definition

>    parse -l sdf -v 2.1 -I -i stratego.def -o stratego.adef
    sdf2sig -i stratego.adef -o stratego.ar
    ast2abox -p /home/visser/res/app/tiger/tmp/front-tiger-0.2/sig/stratego.pp -i  stratego.ar -o stratego.ar.abox
    abox2text -i stratego.ar.abox -o stratego.r
    unpack stratego.r r                         
   less stratego.r                    

=>    Inspecting the signature unveiled a couple of wrong constructor names (forgot about
   Var and SVar)

[Step 18] Added reject rules for reserved words

=>    The performance of generated parsers was not very good and that time was spent
   filtering syntax trees. It occurred to me that I had not declared the reserved 
   words of the Stratego. Added productions of the from |"keyword" -> Id {reject}|
   for all reserved words.

=>    Module names used in import sections are allowed to be reserved words except
   for "rules", "strategies", "signature", and "overlays". Created separate lexical
   sort ModName to reflect this.

[Step 19] Add stratego.0.6.2 to the grammar base.

[Step 20] Project finished. Future work:

=>   Improve the pretty-print table to obtain beautyfier for Stratego specifications.

Introduction

gen-renamed-sdf-module generates an SDF module that renames all SDF sorts in a given SDF definition.

Example

Syntax definition:

definition
module Expressions
imports Identifiers [Id => MyId]
  exports
    sorts Exp

    context-free syntax
      Id        -> Exp {cons("Var")}
      IntConst  -> Exp {cons("Int")}

      Exp "+"  Exp -> Exp  {left, cons("Plus")}

    lexical syntax
      [\ \t\n]  -> LAYOUT
      [0-9]+    -> IntConst

module Identifiers
  exports
    sorts Id

    lexical syntax
      [a-zA-Z]+ -> Id

Invocation:

  > gen-renamed-sdf-module -i Exp.def -m Expressions --name Exp-Prefixed --prefix Exp

Result:

module Exp-Prefixed
imports Expressions
          [ IntConst => ExpIntConst
            MyId     => ExpMyId
            Exp      => ExpExp ]

Introduction

The GPP package is a tool suite for generic pretty-printing. GPP supports pretty-printing of parse-trees in the AsFix format with comment preservation and of abstract syntax trees. GPP supports the output formats plain text, LaTeX, and HTML. Formattings are defined in pretty print tables, which can be generated from SDF syntax definitions.

Tools

• abox2html
• abox2latex
• abox2text

• asfix2abox
• ast2abox

• ppgen
• pptable-diff

Languages

GPP is based on Box, a language independent markup language to describe the intended layout of text.

• Box language
• Pretty Print Table

Documentation

• How to pretty print a grammar
• How to use GPP with LaTeX
• Pretty print tables
• Stratego.StrategoBox

Links

The official homepage of GPP:

• http://www.cs.uu.nl/groups/ST/Merijn/GenericPrettyPrinter

Graph Format Conversion

Currently graph-tools supports the conversion of a subset of GraphXML to Dot.

Other Graph Formats

Other graph formats that might be supported by graph-tools in the future:

• Graph Term Format
  • http://homepages.cwi.nl/~leon/hacks/graph-tools/
• GraphML
  • a comprehensive and easy-to-use file format for graphs
  • is this the succesor of GraphXML?
  • http://graphml.graphdrawing.org
• GXL
  • Graph eXchange Language
  • GXL is designed to be a standard exchange format for graphs. In particular, GXL was developed to enable interoperability between software reengineering tools.
  • http://www.gupro.de/GXL/

Related Packages and Tools

• Graphviz is a package for visualization of graphs created at AT&T (dot is part of Graphviz)